Update: with more tinkering around with logging, i've narrowed it down to the following exception:
java.security.cert.CertPathValidatorException: The CA certificate did not contain BasicConstraints
at com.ibm.security.cert.BasicChecker.check(BasicChecker.java:220)
at com.ibm.security.cert.PKIXCertPathValidatorImpl.engineValidate(PKIXCertPathValidatorImpl.java:289)
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:267)
at com.ibm.domino.napi.ssl.DominoX509TrustManager.checkServerTrusted(DominoX509TrustManager.java:53)
Looks like the root cause is that our server's certificate was signed with a legacy Valicert root certificate (https://certs.godaddy.com/anonymous/repository.pki?streamfilename=valicert_class2_root.crt&actionMethod=anonymous%2Frepository.xhtml%3Arepository.streamFile%28%27%27%29&cid=1381721 this one), and the code in ibmcertpathprovider.jar does not like the fact it does not have a 'Basic Constraints' extension.
I do not see a workaround for this right from top of my head here, maybe you can suggest some solution?